Focused Crawling

Focused crawling is an efficient mechanism for discovering resources of interest on the web. Link structure is an important property of the web that defines its content. In this thesis, FOCUS a novel focused crawler is described, which primarily uses the link structure of the web in its crawling strategy. It uses currently available search engine APIs, provided by Google, to construct a layered web graph. This layered model of the web is used to learn the link that leads to topic pages. It is formulated as an ordinal regression problem that when solved gives link distance of topic pages from any given page. This directly produces an ordering among the links to be crawled. The ordinal nature of the problem removes any need for a negative class which is a problem with the existing crawlers. The large scale nature of the web poses scaling problems to current ordinal regression solvers. To overcome this a novel large scale clustering based ordinal regression solver using Second Order Cone Programming(SOCP) is proposed. Proposed approach is implemented on top of nutch open source crawler, which uses wellknown MapReduce distributed model to make the crawler scalable. Experimental results on different datasets show that the proposed method is competitive.